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EXECUTIVE SUMMARY 



The general purpose of this evaluation is to assess the effectiveness of 
independent charter schools in promoting two desirable student outcomes: 
student achievement growth and educational attainment. Independent 
charter schools are authorized by non-district entities and are considered 
“independent” because they are not a part of the Milwaukee Public School 
District. We will estimate achievement growth of independent charter 
school students in grades 3-8 over four years in reading and math on the 
Wisconsin Knowledge and Concepts Examination (WKCE). Similarly, in 
later reports we will track student attainment, specifically whether upper- 
grade cohorts in our evaluation graduate from high school. Case studies of 
independent charter schools will help us to identify best practices in these 
schools and will also be addressed in future reports. 

This report provides findings comparing the first year of achievement 
growth (2006 to 2007) of students attending independent charters to the 
achievement growth of a group of matched comparison students attending 
Milwaukee Public Schools. Our next report, to be released in spring 2011, 
will examine two- and three-year achievement growth. 

These reports draw upon a panel of all 2,295 students attending 10 of the 
14 independent charter schools in grades 3-8 in 2006-07 with test scores 
for that year. The four charter schools excluded in the sample either were 
not open for both the baseline (2006-07) and outcome (2007-08) years or 
did not enroll students in tested grades. That census of tested Milwaukee 
independent charter school students was then carefully matched to a 
similar-sized panel (2,295) of students attending MPS. 

Using regression models that produce the most precise estimates of 2007 
achievement, our comparisons of students in our sample of independent 
Milwaukee charters to matched MPS students exhibit few significant effects 
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of attending a charter school on achievement growth in either math or 
reading. The exception is in one of our three models for mathematics gains. 
When we control for prior achievement, and not for student characteristics 
or switching schools, students in charter schools gain approximately .105 
standard deviations more in math achievement than students in MPS. 
Further analysis reveals that the positive impact of independent charter 
schools on average in math is concentrated primarily at the lower end of 
the achievement distribution; these schools were estimated to improve the 
math achievement of students at the 25th percentile of the achievement 
distribution by .109 standard deviations. There are no differences in any 
models in reading. 

There are differences, however, when we disaggregate the charter impacts 
by charter school type. Conversion independent charters, schools which 
converted from private schools, hold an advantage in math and reading 
achievement. Prior to controlling for both student characteristics and 
if students switched schools, students in conversion charters make .170 
standard deviations greater gains in math achievement compared to similar 
students in MPS schools. Once controlling for student characteristics 
and school switching, the effect is reduced to .114 standard deviations. 
Similarly, in reading, students in conversion charters make .124 standard 
deviations more gains than MPS students without controlling for student 
characteristics and switching schools. By adding these factors the effect is 
reduced to .054 standard deviations. At the same time, students in non- 
conversion, independent charter schools, schools which began as new 
charter schools or startups, achieve gains that are no different from their 
counterparts in MPS. 

In addition to looking at student performance, we examine the patterns of 
school switching both within-sector and between-sector. We do so because 
school switching tends to disrupt, and therefore negatively affect, student 
learning. In addition, patterns of student switching alter the demographics 
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of schools over time. A greater number of students in MPS switched 
schools than students attending independent charter schools. Similarly, a 
much larger percentage of MPS students are switching for non-structural 
reasons than those in independent charters. These non-structural 
switchers are switching either because they are moving or are potentially 
dissatisfied with their current school. In any case, school switching has a 
negative impact on student achievement gains in our study. 

We caution that the results in this report are based only on the first of 
four years of estimated achievement growth. Subsequent reports may well 
alter the general findings and conclusions. 

We are appreciative of the constructive comments on a preliminary draft 
from outside experts as well as the SCDP Research Advisory Board and 
research team, particularly David Figlio of Northwestern University and 
Brian Gill of Mathematica Policy Research, Inc. Additionally, we thank 
Russell Diamond for advice on data coding. All remaining errors are the 
responsibility of the authors alone. 

This project is being funded by a diverse set of philanthropies including 
the Annie E. Casey, Joyce, Kern Family, Lynde and Harry Bradley, 
Robertson, and Walton Family foundations. We thank them for their 
generous support and acknowledge that the actual content of this report 
is solely the responsibility of the authors and does not necessarily reflect 
any official positions of the various funding organizations, the University 
of Arkansas, or the University of Wisconsin. We also express our gratitude 
to officials at MPS, the independent charter schools, and the state 
Department of Public Instruction for their willing cooperation, advice, 
and assistance. 
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INTRODUCTION 



Charter schools are tuition-free public schools that are authorized to operate within an agreed “charter.” The 
charters often specify the size of the school, its mission, specialized curricula and pedagogy, unique personnel 
practices, and specific goals that the school must meet over time in order to he reauthorized. To facilitate these 
unique schools, they are often given waivers from some of the administrative and accountability requirements of 
other public schools. This does not exempt charter schools from the requirements of the federal No Child Left 
Behind law. 

Since the opening of the first charter schools in the early 1990s, charter schools have grown widely. In 2009, 
5,043 charter schools served close to 1.5 million students in 40 states and the District of Columbia (Center for 
Education Reform 2009). Following the recent “Race to the Top” initiative, which requires states to relax charter 
school laws to be competitive for federal education funds, we can expect to see even further expansion of charter 
schools in the coming years. 

Supporters see the potential of high-quality charter schools to help transform the education system by raising 
achievement levels, closing achievement gaps, providing competitive pressure to traditional public schools 
and stimulating greater innovation. They posit that giving charter schools more flexibility over such practices 
as hiring teachers, budgeting school funds, and selecting curricula will lead to these positive outcomes (Finn, 
Manno and Vanourek 2001; Payne and Knowles 2009). Further, through a system of accountability, they expect 
to reduce the number of low-quality charter schools that are not able to meet the standards they agreed to in 
their charters. 

In contrast, critics are concerned about charter schools drawing away resources from traditional public schools 
(e.g. teachers, funding, and motivated students), increasing racial segregation, and lacking the accountability 
structure to close or improve low-quality charter schools (Wells et al. 2002). They fear charters are performing no 
better and sometimes worse than traditional public schools. To date the research on the performance of charter 
schools is mixed, ranging from negative, neutral, mildly positive, to a few specific studies which are strongly 
positive (Bifulco and Ladd 2006; Sass 2006; Ballou et al. 2006; Hanushek et al. 2007; Booker et al. 2007; 
Zimmer et al. 2009; Witte et al. 2007; Witte and Lavertu 2009; CREDO 2009; Hoxby et al. 2009; CREDO 
2010; Abdulkadiroglu et al. 2009; Tuttle et al. 2010; Gleason et al. 2010). 

This longitudinal evaluation will evaluate the impacts of independent charter schools on student achievement 
in Milwaukee, Wisconsin. Milwaukee is one of the few places in the U.S. that contains both district-authorized 
charter schools and independent charter schools (Table 1). As of the beginning of this study, the 35 district- 
authorized charter schools remain part of the Milwaukee Public School system. Of these 35 district-authorized 
charters, a total of 26 are staffed by teachers who remain employees of the school district and bound by the 
union-negotiated collective bargaining agreement. These schools are referred to as “instrumentality” charters. 
The remaining nine MPS “non-instrumentality” charter schools are permitted to hire and employ non-union 
teachers. Of the 14 Milwaukee public charter schools that operated independently of the district in 2006- 
07, nine were authorized by the University of Wisconsin-Milwaukee and five were authorized by the City 
of Milwaukee. 
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Type 


Number 


Percentage 
of All 


MPS Instrumentality 


26 


53.1 


MPS Non-Instrumentality 


9 


18.4 


MPS Total 


35 


71.4 


Independent U of W-Milwaukee 


9 


18.4 


Independent City of Milwaukee 


5 


10.2 


Independent Total 


14 


28.6 



While there has heen prior research comparing student outcomes in the MPS instrumentality and non- 
instrumentality charter schools to those in traditional public schools (Witte and Lavertu 2009; Lavertu and 
Witte 2008; Witte et al. 2007), researchers have yet to evaluate the effectiveness of the population of independent 
public charter schools in the city. Initially using two years of panel data from 2006-07 and 2007-08, we will 
estimate growth models of student gains controlling for student characteristics and school switching. We 
estimate the impact of independent charter attendance on student achievement growth in reading and math for 
students in grades 3 through 8 during the baseline year of 2006-07. 

We also include in this study a descriptive analysis of students who switch schools and/or move from traditional 
MPS schools to independent charter schools and vice-versa. School switching can change the composition of 
charter schools and complicate the analysis of charter effects, as charter school students move to traditional 
public schools and vice-versa. In addition, we identify those students who are missing from our samples 
(attrition) after the first year. Study attrition can bias any longitudinal analysis to the extent that the charter 
school students who “disappear” are systemically different from the comparison group students who leave the 
study. We describe these conditions in this report to demonstrate that they are not yet serious concerns. 

Data access permitting, all of these analyses will be continued for three more years. In addition, we hope in the 
future to analyze graduation from high school for charter and non-charter students, and, through case studies, 
provide insights into what types of schools and educational practices seem to work best within both charter and 
traditional public schools. 

Prior Research on Charter School Performance 

The literature on charter school performance focuses on three major areas: achievement, competition, and 
segregation. To determine the effectiveness of charter schools, the majority of studies evaluate the achievement 
of students in charter schools on standardized tests relative to students in traditional public schools. Fewer 
studies consider the impact of charter school policy on the academic outcomes of traditional public school 
students through competition, or how they may influence the sorting of students into environments that are 
either more diverse or racially isolated. Since student outcomes for charter students are the focus of this study, 
we briefly describe just the first set of studies - those on student achievement. Our description is limited to a 



Milwaukee Independent Charter Schools Study: Report on One Year of Student Growth 





December 2010 



set of meta-analyses across charter school studies and then a brief description of the two prior studies of charters 
authorized hy the Milwaukee Public School system. 

The first evaluations of charter school achievement seemed to suggest that charter schools performed no better 
than traditional public schools, on average. The Charter Schools Dust-Up, a meta-analysis of early charter schools 
studies, found that students in charter schools scored about the same or sometimes worse on standardized tests 
compared to students in traditional public schools (Carnoy et al. 2005). However, more recent reviews of panel 
studies evaluating charter school achievement contain findings which suggest results are more mixed and more 
positive than the findings of Carnoy et al. (2005). 

The Carnoy et al. study was followed by another meta-analysis by Hill, Angel and Christensen (2006). They 
initially reviewed 35 charter school achievement studies that were published since 2000. In their review, they 
found that the impact of charter schools on student achievement is null or mixed in most published studies. 
These studies ranged in methodological quality and for detailed analyses they only relied on the results of the 
five most sophisticated studies from states with good longitudinal data: Solomon and Goldschmidt (2004) in 
Arizona, Hanushek et al (2005) in Texas, Sass (2006) in Florida, Bifulco and Ladd (2006) in North Carolina, 
and Booker et al. (2004) in Texas. From these studies, there were two positive, two mixed and one negative 
results. Where there were positive or negative findings, the results were small in magnitude. The authors 
interpret the results from their review as preliminary evidence and remark that there was not yet enough 
research to definitively account for the impact of charter schools on student achievement. 

Betts and Tang (2008) analyzed 14 rigorous studies that used either value-added longitudinal methods or 
lotteries. Their results were also mixed, but the overall results were on the positive side in terms of the number 
of studies and the effect sizes. However, there was considerable variation by grade level. They found charter 
schools did particularly well in reading at the elementary level. On the other hand, the results were more likely 
to be negative for charter high schools in contrast to traditional high schools. 

Even more recently, Nicotera (2009) reviewed 140 studies and reports on 33 panel studies of charter school 
effectiveness. Of 81 findings regarding math achievement and 79 findings regarding reading achievement she 
reported that the findings are almost equally distributed across the three categories of “charter advantage,” “no 
difference,” and “traditional public school advantage,” except that a plurality of the math achievement findings 
favored traditional public schools. However, charter effectiveness studies drawing upon data prior to 2001 
overwhelmingly reported a traditional public school advantage; whereas, a plurality of studies drawing upon 
more recent data indicated a charter school advantage. 

Two other recent national studies of public charter school performance similarly come to complex conclusions. 
The Center for Research on Education Outcomes (CREDO) program at Stanford University examined 
charter students in over 2,400 schools in 15 states and the District of Columbia, matching each charter student 
to a “virtual twin” based on demographic and baseline performance characteristics (Center for Research on 
Education Outcomes 2009). Three years later, researchers reported that 17 percent of the charter schools had 
increased test scores relative to their comparison traditional public schools, 46 percent generated no significant 
difference, and 37 percent of the charter schools decreased math scores. The low-income and English Language 
Learner subpopulations of students demonstrated the clearest test score benefits from attending charter schools, 
and students tended to gain more the longer they stayed in their charter school. 
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Researchers at Mathematica Policy Research took advantage of lotteries at 36 charter middle schools in 15 states 
to conduct an experimental analysis of charter impacts. Three years after the lottery, the students offered admission 
to a charter middle school were performing, on average, similarly to the students who were not offered admission 
(Gleason et al. 2010). As with the CREDO study, the disaggregated results revealed interesting patterns. The urban 
charter schools in the study produced higher achievement gains in math for their students, compared to the control 
group, while the rural charter schools in the study generated relative achievement losses for their students. Low- 
income and low-achieving students gained more in achievement if they won their charter school admissions lottery. 

Mathematica Policy Research also recently released an evaluation of 22 middle schools that are specifically part 
of the Knowledge is Power Period (KIPP) network of independent charter schools (Tuttle et al. 2010). Using a 
matched longitudinal panel approach similar to the CREDO study, the researchers report that in half of the KIPP 
schools effect sizes in math (0.48) and reading (0.28) represent 1.2 years and .9 years extra accumulated growth, 
respectively, for charter school students in a three year period. 

Research on charter schools authorized hy the Milwaukee Public School District showed students in these schools 
made modestly greater gains compared to traditional public school students. The first of these studies, Witte et al. 
(2007), examined the impact of charter status on achievement and student proficiency levels over several years using 
student fixed effects and difference-in-difference regression and logistic regression models. They found that MPS 
charters were performing a bit better than traditional public schools. Similarly, in a more recent paper using more 
years of data, Witte and Lavertu (2009) found larger gains for MPS charter school students in Milwaukee than 
for traditional public school students in math but not reading. These two prior studies seem to suggest that MPS 
charter schools are having a positive impact on the students who attend them. However, none of these prior studies 
evaluate data from independent charter schools because the data were previously unavailable. 

The Milwaukee Context: Charter School Reform and Independent Charters 

Similar to national trends, the number of charter schools in Wisconsin has grown widely from 17 in 1997 to 206 
in 2009 (Evers et al. 2009). Charter schools serve more than 37,000 students in the state (Center for Education 
Reform 2009; Evers et al. 2009). Government officials see the potential of charter schools as part of a reform to 
transform public education in the state. For the first time. Governor Jim Doyle and State Superintendent Tony 
Evers attended the Wisconsin Charter Schools Conference in April, 2009 (Borsuk 2009). In October of 2009, 
President Barack Obama and Education Secretary Arne Duncan visited Wright Middle School, a charter school in 
Madison, to highlight the role of charter schools in the “Race to the Top” initiative. Most recently, Wisconsin has 
also received $86 million in federal funding over the next five years to support charter schools in Milwaukee and the 
state by allocating grants to new and existing charter schools. 

In Milwaukee, charter schools are one among a wide variety of school choice options including charter and magnet 
schools affiliated with MPS, open enrollment into other public school districts, and private schools accepting 
vouchers under the Milwaukee Parental Choice Program. In 2006-07, charter schools in Milwaukee comprised 
close to a quarter of the charter schools in the state. As discussed previously, independent charters are a distinctive 
type of charter school in Milwaukee. They were created by 1997 legislation to be authorized by the City of 
Milwaukee Common Council, the University ofWisconsin-Milwaukee (UWM), the Milwaukee Area Technical 
College, or the University of Wisconsin at Parkside (Racine). They are not connected to MPS. Of the 9 UWM 
and 5 City of Milwaukee independent charters open in 2006-07, 10 are the subjects of this research. The student 
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enrollments by grade for the baseline year of 2006-07 for our school sample are indicated in Table 22 As is 
apparent, UWM charter schools have many more students than City charters, and there are very few students in 
grade 9 compared to grades 3 to 8. 



Table 2. 

Milwaukee Independent Charter School Sample Enrollment, 2006-07 



Grade 


Schools 


3 


4 


5 


6 


7 


8 


9 


TOTAL 


UWM 


6 


328 


331 


338 


287 


241 


239 


140 


1,904 


City 


4 


92 


89 


99 


88 


119 


80 


58 


625 


TOTAL 


10 


420 


420 


437 


375 


360 


319 


198 


2,529 



Source: Charter Schools page on the Department of Public Instruction website: 
http://www.dpi.state.wi.us/sms/xls/0607enrl.htm 

Note: For the 2006-07 baseline year, there are no test score data for structural reasons for the following schools: Inlands 
Sea School of Expeditionary Learning (ISSEL), Milwaukee Renaissance Academy (MRA), Seeds of Flealth (SoFIE), School 
for Early Development (SEDA), and Massai Institute which has closed as of 2007-08. For the first four schools, they did 
not test in November 2006 when schools typically test because they were not yet open. ISSEL opened in 01/2006, MRA 
in 08/2007, and SoFIE in 08/2007. SEDA is an early education school with grade levels K4-2 and does not have data for 
grades 3-8, or grade 10 because it does not have these grade levels at its school. Bruce Guadalupe Community School 
transitioned from the oversight of MPS to being authorized as a charter by UWM in 2009-2010. In addition, in the current 
2010-2011 school year, the City chartered King's Academy while UWM chartered Urban Day School, Veritas Fligh School 
and a new campus of Milwaukee College Preparatory School, Lindsay Fleights. Many of the students attending Lindsay 
Fleights formerly attended the Academy of Learning and Leadership which closed in September 2010. 



RESEARCH QUESTIONS AND METHODOLOGY 

Through this evaluation we expect to understand whether students benefit in the short term and the long term 
from attending an independent charter school. We are interested in both educational achievement and attainment, 
as measured by grade retention and most importantly by high school graduation. Over the next three years we are 
also interested in evaluating what appear to be the keys to successful charter schools. We will report on attainment 
and best practices in future reports. In this report the primary research question is: Do Milwaukee’s independent 
charter schools produce higher rates of student learning growth, over the short term, than do Milwaukee Public 
Schools? For purposes of this study, achievement is measured by performance on the reading and math sections 
of the Wisconsin Knowledge and Concepts Examination (WKCE) that all public school students are required 
to take in grades 3 to 8 and 10. The WKCE is administered in November of each school year. In the WKCE 
students are evaluated through short answer and multiple choice questions about their mastery in reading, math. 



1 The number of students with third grade test scores in Table 3 differs from the enrollment number because the data are collected 
at two different times. The enrollment data are based on the September 5th enrollment count. When test scores are added into 
the enrollment file in November, it is very likely that additional students enrolled in the school. 
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Figure 1: Reading Comparisons (2006-2007) 



• Independent 
Charters 

- MPS Matched 
. MPS Random 



language arts, science and social studies. Scores on these examinations are recorded in both scale (or developmental) 
scores and proficiency levels. We rely for part of our analysis on scale scores. As indicated below, at times we 
standardize these scale scores so they can be compared across grade levels. The first step in our analysis was to 
determine the comparative samples of students. Because the total number of students in independent charters for 
which test scores were available in 2006-07 was 2,295, we decided to include all of those students in the charter 
school sample. The issue was then how to create a relevant matched sample that would be similar on important 
observed characteristics at baseline. To do that we first selected a random sample of MPS students matched by 
grade. In doing so we discovered that the baseline test scores (November 2006) for that group differed from those 
in the independent charter schools in a number of grades. As depicted in Figures 1 and 2, the random MPS sample 

of students usually scored higher in 2006 than 
the independent charter students. Thus the 
random sample would have started out students 
at different levels of prior achievement. 

To adjust for this problem we undertook a two- 
step procedure. First, each student in the charter 
panel was matched with the set of MPS students 
in their grade with baseline WKCE test scores 
within five percent of their score. There were 20 
such bands with the lowest being scores from 
the first to fifth percentile, and the highest for 
those scoring from 95% to 100%. Second, the 
charter panelist and each MPS student within 
that five percent “grade band” were assigned a 
“propensity score” that predicted their likelihood 
of being in a charter school based on race, gender, 
English Language Learner (ELL) status, and 
participation in the federal Free/Reduced Price 
Lunch (FRL) program. The MPS student within 
the grade band with the charter school propensity 
score closest to the propensity score of a given 
student in the charter panel was drawn out of the 
panel (without replacement) to serve on the MPS 
comparison panel. The result of all these matches 
was a panel of 2,295 independent charter school 
students and 2,295 MPS comparison students 
that closely resemble the charter school students 
on baseline test scores and other factors that 
predict charter school enrollment. 

The purpose of this procedure was to reduce the 
differences in observed characteristics between 
Grade the independent charter students and a random 




Grade 



Figure 2: Math Comparisons (2006-2007) 




• Independent 
Charters 

>MPS Matched 
.MPS Random 
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sample of MPS students. Figures 1 and 2 and Table 3 depict our success in improving on the extent to which 
our charter and comparison groups are carefully matched. These figures and table display mean comparisons by 
grade level for the three possible comparison samples. 

In Figure 1, for reading, the Independent Charter Sample and MPS Matched Sample begin at similar points 
and converge in later grades. While there appears to be a gap over grades 5 through 7, these mean differences 
between the two sectors are not statistically significant. For math, in Figure 2, the charter and MPS matched 
samples are almost indistinguishable at all grades. 

In Table 3, we see few statistically significant differences between means in reading and math scale scores when 
comparing the Independent Charter Sample and the MPS Matched Sample. With the exceptions of 4th and 
6th grade math, this holds for every grade in both subjects. Both of those lingering differences between the 
charter and matched samples are statistically significant only at the 90 percent confidence level, the lowest 
confidence level that we use in this evaluation. This suggests the matching was successful. Thus, in terms of prior 
achievement we have created the proverbial apples-to-apples comparison. 



Tables. 

Baseline (2006-07) Scale Scores of Independent Charter, MPS Matched, and MPS Random Samples 



Grade 


Sample 


N 


Mean 
Reading 
Scale Score 


Reading 

SD 


Mean Math 
Scale Score 


Math 

SD 


Comb Mean 
Scale Score 


Combined 

SD 


3 


Independent 


430 


429 


47 


385 


49 


407 


44 


3 


MPS Matched 


431 


427 


47 


383 


47 


405 


42 


3 


MPS Random 


341 


440*** 


38 


402*** 


50 


422*** 


40 


4 


Independent 


417 


441 


49 


421 


48 


431 


45 


4 


MPS Matched 


420 


440 


53 


415* 


54 


428 


50 


4 


MPS Random 


324 


447 


53 


434*** 


47 


441*** 


47 


5 


Independent 


427 


447 


51 


452 


48 


450 


46 


5 


MPS Matched 


430 


451 


49 


454 


41 


453 


40 


5 


MPS Random 


338 


448 


56 


452 


44 


451 


47 


6 


Independent 


358 


460 


51 


473 


48 


466 


46 


6 


MPS Matched 


356 


464 


50 


467* 


40 


466 


41 


6 


MPS Random 


330 


464 


54 


469 


44 


468 


44 


7 


Independent 


346 


470 


51 


493 


43 


482 


44 


7 


MPS Matched 


341 


470 


45 


496 


38 


483 


38 


7 


MPS Random 


303 


468 


49 


499 


47 


488* 


44 


8 


Independent 


298 


499 


47 


507 


48 


503 


43 


8 


MPS Matched 


299 


499 


50 


506 


45 


503 


43 


8 


MPS Random 


290 


488** 


55 


497** 


50 


493** 


53 



***Different from Independent Charter at p<0.01, **Different from Independent Charter at p<0.05, 
*Different from Independent Charter at p<0.10 
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Table 4 indicates that our matching algorithm was successful in producing a matched sample with important 
measurable student characteristics that, though imperfect, are closer than in the random sample. The 
matched and charter samples are very close on race and gender. Although the matched MPS sample contains 
populations of exceptional education and free lunch students that are closer to the charter sample than the 
random sample provided, the MPS Matched sample does differ significantly from the Independent Charter 
Sample regarding these two student characteristics. Because of these differences, in most of the analyses to 
follow we independently control for all of these student characteristics in our regression models.^ 

Table 4. 

Comparison of Student Characteristics in Three Possible Study Samples 



Sample 


Black 

(%) 


White 

(%) 


Hispanic 

(%) 


Female 

(%) 


ExEd 

(%) 


Federal 

Lunch 

(%) 


ELL 

(%) 


Independent Charter 


93.93 


3.86 


1.80 


49.67 


8.38 


51.64 


0.04 


MPS Matched 


98.19*** 


1 44 *** 


0.26*** 


49.45 


16.07*** 


73.82*** 


0.04 


MPS Random 


59.91*** 


14.52*** 


19.76*** 


48.29 


19.47*** 


83.68*** 


13.27*** 



Stars indicate MPS Matched and MPS Random different from Independent Charter at 
***p<0.01, **p<0.05, *p<0.10, based on a two-tailed t-test. 



Our matching protocol and baseline control variables will limit the extent to which measurable student 
characteristics might bias our analysis of independent charter and MPS student test score gains. Because 
students were not randomly assigned to the two groups, however, we cannot rule out unmeasured student 
characteristics as a potential source of bias. For example, if the students in Milwaukee independent charter 
schools are similar to our matched MPS sample in most ways except that they have more motivated parents, as 
demonstrated by the fact that they enrolled the student in a school of choice, then the charter students might 
demonstrate stronger achievement gains simply due to such a “self-selection” bias. On the other hand, if parents 
seek alternatives to their neighborhood public school primarily when their child is struggling, the match on 
baseline achievement might not fully capture the inherent educational disadvantages of charter school students, 
thereby biasing our analysis against better performance from charter schools. 

We think that the fact that our study is situated in Milwaukee helps to reduce the threat of positive or negative 
unmeasured selection biases. As discussed above, many school choice options are available to parents even 
within the Milwaukee Public School (MPS) system. Highly motivated parents, or parents of students who are 
struggling in their neighborhood public school, can and likely do seek out alternative placements for their child 



2 The initial difference between the charter and matched sample on free lunch status is due to incomplete free lunch data counts 
in a few schools. We correct for this in our models in two ways. If a student had a free lunch observation in 2007-08, 2008-09 
or 2009-10, we backfilled the data. In addition, for students with missing data on free-lunch or any other control variable, we 
include an indicator in our models controlling for this missing data. Doing so allows our regression models to draw upon the 
actual data in each student observation, and only that actual data, to inform the coefficient estimates of the model (Cohen and 
Cohen 1983). 
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within MPS. Since school choosers are present in hoth our charter and MPS matched comparison samples, 
concerns about self-selection bias when comparing student achievement gains across sectors are, to some 
extent, mitigated. 



STUDENT ACHIEVEMENT GAINS: 2006-07 to 2007-08 



Average Math and Reading Achievement 

We employ descriptive statistics and multivariate methods to compare one year gains for students in 
independent charter schools and comparable, matched students in Milwaukee Public Schools. We first analyze 
scale scores that increase in mean and range in each successive grade as more subjects are covered in each test. 
These scores have excellent psychometric properties but do not allow direct comparisons across grades or direct 
understanding of effect sizes. For these reasons we construct standardized z scores from scale scores using the 
MPS district means and standard deviations for math and reading. For all MPS students this procedure would 
produce an average z-score of 0 with a standard deviation of 1.0.^ Our samples may deviate from these norms at 
baseline and subsequently those data are relevant findings. These normalized z scores are used in the analyses in 
Tables 5 to 13. 

In Tables 5 and 6, we report the average gains for each sample and differences in gains in math and reading 
between the samples. The results of this first analysis are broken out by grade level to examine the variation in 
student learning gains by school type across the different grades. In general, the grade-specific results are a mix 
of positive charter effects and no significant differences. Fifth grade charter school students gained an average 
of .111 standard deviations in math achievement compared to matched 5th graders in MPS (Table 5). Sixth 
grade charter school students gained an average of .320 standard deviations more in math than similar MPS 
students. Both of those positive charter school effects on math gains were statistically significant at high levels. 
The differences in math achievement gains for charter versus matched MPS students in 4th, 7th, and 8th grades 
were not statistically significant. Because of the gains in grades 5 and 6, the total effect for all grades is about 
.093 standard deviations higher for charter school students. 

In reading, the gains for 6th grade charter students were an average of .148 standard deviations higher than their 
matched MPS counterparts, and the charter school advantage in reading gains in 7th grade was an average of 
.105 standard deviations (Table 6). Both of those differences were statistically significant at high confidence 
levels. No statistically significant gain score differences in reading were observed between charter and matched 
MPS students in 4th, 5th, or 8th grade. 



3 We computed normalized z scores by grade level in both years for reading and math. For example, the formula for ZMath2007 
in Grade 3 would be ((Grade 3 ScaleMath2007- Grade 3 MPS district mean scale score)/(Grade 3 MPS district standard 
deviation)). 
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Table 5. 

Standardized Mean Math Achievement by Grade, 2006-07 to 2007-08 



Grade 2007 


Group 


Average Math 
Gains 2006-2007 
(Change) 


s.e.(diff) 


4 


Charter 


.032 






MPS Matched 


.031 






(Difference) 


(.001) 


.055 


5 


Charter 


.216 






MPS Matched 


.105 






(Difference) 


(.111)** 


.052 


6 


Charter 


.121 






MPS Matched 


-.199 






(Difference) 


(.320)*** 


.056 


7 


Charter 


-.048 






MPS Matched 


-.063 






(Difference) 


(.015) 


.054 


8 


Charter 


-.036 






MPS Matched 


-.113 






(Difference) 


(.077) 


.060 


All Grades 


Charter 


.059 






MPS Matched 


-.034 






(Difference) 


(.093)*** 


.025 



***p<0.01, **p<0.05, *p<0.10 



Note: Two sample t-tests were run to test the significance of differences in average gains between our MPS Matched 
sample and Charter sample. Response weights were included in the estimation of differences in means. 
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Table 6. 

Standardized Mean Reading Achievement by Grade, 2006-07 to 2007-08 



Grade 2007 


Group 


Average Reading 
Gains 2006-2007 
(Change) 


s.e.(diff) 


4 


Charter 


-.005 






MPS Matched 


.003 






(Difference) 


(-.008) 


.059 


5 


Charter 


.012 






MPS Matched 


.022 






(Difference) 


(-.010) 


.052 


6 


Charter 


.092 






MPS Matched 


-.056 






(Difference) 


(.148)*** 


.052 


7 


Charter 


.063 






MPS Matched 


-.042 






(Difference) 


(.105)** 


.049 


8 


Charter 


.031 






MPS Matched 


.001 






(Difference) 


(.031) 


.056 


All Grades 


Charter 


.034 






MPS Matched 


-.008 






(Difference) 


(.042)** 


.025 



***p<0.01, **p<0.05, *p<0.10 



Note: Two sample t-tests were run to test the significance of differences in average gains between our MPS Matched 
sample and Charter sample. In some cases, the changes in means may not sum exactly because of rounding. 
Response weights were included in the estimation of differences in means. 
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The Distribution of Math and Reading Growth 

It is equally as important to understand the distribution of gains as it is to note averages. The “density,” which 
approximates the number of individuals at specific points in the achievement distribution, is provided in “kernel” 
diagrams in Figures 3 and 4. Examining the kernel density distributions of both sectors in reading and math 
will help us to understand the shape of the growth in our samples. For both reading and math, the distributions 
are mostly overlapping and are normally distributed. If we were to see non-overlapping distributions this would 
indicate a great variation in performance between the two sectors. For math, MPS matched students have 
somewhat more students just below the middle of the distribution, while the charter school distribution has 
more students to the right of the mean. In reading (Figure 4) there is a slight advantage for charter students 
above the mean. The range of students scoring at the high end and low end are very similar in both sectors. 

Figure 3: Figure 4: 

November 2007 Standardized Math Growth November 2007 Standardized Reading Growth 

for All Students in Grades 4-8 for All Students in Grades 4-8 



Math Growth 2006-2007 



Independent Charter 
UPS Matched 




•SOS 
ZMalh2007 • ZM«h2006 

« •panacftnitov bgxhwJlh • 0 1037 



Reading Growth 2006-2007 

Indepeodeni Chanar 




- 



^•2 0 2 4 

ZRMdtn^r • ZReedm92006 

' eeMcrviao* bdnpMMdm < 0 100» 



We further examine the gains of independent charter school students compared to students in MPS in reading 
and math with a non-parametric measure called Somers’ D.'* With this ordinal measure we calculate the 
difference in the probability that a given independent charter school student will demonstrate more or less 
gains than a matched MPS student. Positive and significant values for independent charter school students 
indicate they are making more gains than their MPS counterparts. As indicated in Table 7, the results of 



4 Following the procedure employed by Witte et al. 2010 to compute the Somers' D statistic, we first compared the growth of two 
given students (e.g. a comparison) in the two samples. Then, if the student was in MPS and achieved greater growth they were 
assigned a score of -1. Similarly, a charter school student who achieved greater growth was assigned a score of +1. If each of the 
students exhibited the same growth they were assigned a score of 0. The scores from each of these comparisons were summed 
and then divided by the total number of comparisons to get the Somers' D coefficient. 
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this quite different approach largely confirm the difference of mean results indicated in Tables 5 and 6 . There 
are significant advantages for charter students overall in both math and reading, with the math gains most 
prominent in grades 5 and 6 and the reading gains in grades 6 and 7. 

Table?. 

Somers' D Statistics for Math and Reading Growth: 2006-07 to 2007-08 



Subject/Grade 


Somers' D Coeff. (s.e.) 


Math 4 


,006(.043) 


Maths 


,100(.043)** 


Math 6 


,236(.041)*** 


Math 7 


,048(.046) 


Maths 


.053 (.049) 


Math All Grades 


.086(.020)*** 



Subject/Grade 


Somers' D Coeff. (s.e.) 


Reading 4 


.027(.043) 


Reading 5 


-.000(.043) 


Reading 6 


.128(.043)** 


Reading 7 


.114(.046)** 


Reading 8 


.032(.049) 


Reading All Grades 


.056(.020)** 



***p<0.01, **p<0.05, *p<0.10, two-tailed. Results unweighted 



Models for Math and Reading Achievement 

Using an analytic sample of 3,357 students in grades 3 to 7 in the independent charter and MPS matched 
sectors during the 2006-07 school year, we estimate the impact of independent charter school attendance on 
growth in reading and math controlling for student characteristics and switching schools.^ To control for 
potential differences by grade, we include grade indicator variables in all equations. We control for baseline 
achievement by including the student’s prior year (2006) test scores in both subjects. The basic model is 
represented by the following equation: 



(eq 1 ) 



Yaooy.r Po + + ^ 3 ^ 2006 .,+ + P^Sch. + s. 



In this equation for each student i, represents the effect of student enrollment in a charter school in 2006-07 
(C=l) and P 2 and P 3 estimate the impact of baseline math and reading achievement. With this specification, the 
contribution of the baseline test to the estimate of the second year test is unconstrained in that P^ and P^ can 



5 We had 1,179 missing test scores for 2007-08 in math and 1,182 in reading. For students who switched sectors after taking 
tests in November 2006, if we could locate their tests in 2007 in the new sector, we included them in the analysis attributing 
their growth results to their initial sector placement. This is standard practice for "crossovers" in randomized field trials. Also 
in this case it is safe to conclude that most of the sector switching took place over the summer, thus the majority of learning 
occurred for this first year in their initial sector. In subsequent years, as crossover enrollment increases, we will handle analysis 
of crossovers in multiple ways. See Witte etal. 2010. 
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take any value. represents a vector of grade-specific contributions to the intercept (especially important in 
this context where the dependent variable by design increases by grade). In addition, P^ represents the impact 
of a set of student-level characteristics, X, such as gender and race/ ethnicity, and P^. represents the impact of 
switching schools either within the same sector or between sectors (Sch.=l).^ 

In addition to understanding the main effect of attending an independent charter or a traditional school, we are 
also interested in two different types of charter schools. Some of the independent charter schools were initially 
private schools that changed school sectors by converting to public-school charters (i.e. conversion charters). 
Other charters were either startup schools or former public schools (i.e. non-conversion charters). We capture 
and test for the differential effects of these two types by estimating equation 2. 



(eq 2 ) 



Y,007,r Po - PrCC> P,NCC> . p,X,. . p,Sch^. . a,. 



In this specification we split the charter indicator variable in equation 1 into conversion charters (CC) and non- 
conversion charters (NCC), with the effects captured by estimating the P^and P^ parameters. 

The outcomes of interest are 2007-2008 reading and math scale scores taken from the Wisconsin Knowledge 
Concepts Examination (WKCE) in grades 4 to 8. Student characteristics included are those typically found 
in studies of charter school performance and they include free and reduced lunch status, exceptional education 
status (ExEd), race, and gender. English language learner status was not included because there were very 
few ELL students in the charter schools. The race indicator is coded as 1 for black and 0 for non-black as the 
reference group. We collapse racial groups other than blacks into the non-black category because there are 
substantially fewer whites, Hispanics, Asians, and Native Americans in the sample® Because there were some 
differences in the matched samples on baseline, we applied weights to cases based on their inverse proportions 
regarding student characteristic variables. Students with atypical characteristics were weighted more and 
students with typical characteristics were weighted less. However, the weighted results reported below are 
substantively similar to the results from estimating the same models on unweighted samples. The models 
account for the clustering of students within schools and employ robust standard errors. 



6 Some researchers have used differences in test scores as the dependent variable by subtracting the first year test score from 
the second. However, if we want to model achievement growth controlling for prior achievement, this has the effect of 
constraining the effect of prior achievement to 1 .0, which empirically is not the true parameter. Thus, we favor the estimation 
model in Equation 1. 

7 We include all switching between schools including switches that may occur for "structural" reasons, meaning switches that 
have to be made because the student is at the terminal grade in the school. This is done because other research has shown 
that all switches have an impact on tests taken in the year the switch is made. See Zimmer eta/. 2009. Further, although 
one could argue that structural switching is likely to be part of the charter program and should not be controlled, we cannot 
definitively determine if such switches were program related (i.e. charter slots not available or charter schools deny) from 
residential or other reasons for changing schools. 

8 There are initially 3,789 blacks and 144 non-blacks in the sample. 
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Our last set of models examines potential variation in charter school impacts using quantile regressions. We 
include these models because the charter impacts on student learning gains may not he constant among students 
with different levels of achievement. Charter impacts are examined at the 10*, 25*, 50*, 75* and 90* percentiles 
of the baseline achievement distribution. All of the controls described for the previous set of models are included 
in these models. For the quantile regressions, bootstrapped standard errors are estimated and account for the 
clustering of students within schools. 

Results for Models of Charter Impacts on Math and Reading Achievement, 2006-07 to 2007-08 



Tables. 

Models of Independent Charter Sector Impacts on Math Achievement, 2006-07 to 2007-08 



Model 


1 


2 


3 




Est. (s.e.) 


Est. (s.e.) 


Est. (s.e.) 


Charter 2006 


.105(.049)** 


.073 (.047) 


.055(.047) 


2006 Score- Math 


.609(.027)*** 


.606(.025)*** 


.603(.025)*** 


2006 Score- Reading 


.211 (.021)*** 


.162(.020)*** 


.161 (.020)*** 


Black 




-.270(.051)*** 


-.269(.051)*** 


Female 




.012(.027) 


.009(.027) 


ExEd 




-.337(.058)*** 


-.341 (.059)*** 


Free_Lunch 




-.019(.025) 


-.015(.025) 


Switch Sch. 






-.074(.033)** 


Intercept 


-.053(.022)** 


.261 (.060)*** 


.284(.060)*** 


R' 


.572 


.586 


.588 


F 


879.23 






N 


3357 


3357 


3357 



***p<0.01, **p<0.05, *p<0.10 



Note: Test scores measured as standardized scores. Model estimated with robust standard errors. Most of the 
variabies included in the modeis are indicator variables. The reference category for both conversion charter and non- 
conversion charter is MPS. For the biack indicator variabie the reference category is non-biack which inciudes whites, 
Asians, Fiispanics and Native Americans. Maie is the reference category for femaie. For ExEd, Free_Lunch and Switch 
School variables the reference categories are not indicating this status. Grade level dummies and indicators for missing 
values of Free_Lunch and ExEd are not included in the above estimates. 
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Table 9. 

Models of Independent Charter Sector Impacts on 
Reading Achievement, 2006-07 to 2007-08 



Model 


1 


2 


3 




Est. (s.e.) 


Est. (s.e.) 


Est. (s.e.) 


Charter 2006 


.037(.033) 


.007(.029) 


-.012(.029) 


2006 Score- Reading 


.554(.023)*** 


.500(.025)*** 


.499(.025)*** 


2006 Score- Math 


.253(.022)*** 


.246( .020)^** 


.243(.021)*** 


Black 




-.165(.044)*** 


-.164(.046)*** 


Female 




.060(.025)** 


.057(.026)** 


ExEd 




-.316(.057)*** 


-.321 (.057)*** 


Free_Lunch 




-.129(.033) 


-.124(.033)*** 


Switch Sch. 






-.081 (.031)** 


Intercept 


.002(m0}*** 


.279(.054)*** 


.306(.058)*** 


R^ 


.574 


.590 


.591 


F 


558.85 






N 


3354 


3354 


3354 



***p<0.01, **p<0.05, *p<0.10 

Note: Test scores measured as standardized scores. Models estimated with robust standard errors. Most of the 
variables included in the models are indicator variables. The reference category for both conversion charter and non- 
conversion charter is MRS. For the black indicator variable the reference category is non-black which includes whites, 
Asians, Flispanics and Native Americans. Male is the reference category for female. For ExEd, Free_Lunch and Switch 
School variables the reference categories are not indicating this status. Grade level dummies and indicators for missing 
values of Free_Lunch and ExEd are not included in the above estimates. 



First year results comparing the gains of students in independent charter schools to those in MPS are mixed. 

For most of the main effects for reading and math in Tables 8 and 9, the charter coefficients are not significant 
at conventional levels of statistical significance. The exception is for math when we just include prior test scores 
in the estimation model (Table 8, column 1). In that specification, charter school students gain approximately 
.105 standard deviations more than MPS students. However, when we include student characteristics (Table 8, 
column 2) and then student characteristics and school switching (Table 8, column 3) as control variables, charter 
school students do no better in math than the matched MPS students. As indicated in Table 9, there are no 
significant differences in reading for charter schools under any estimation. 

Consistent with prior research, students in both the MPS and independent charter sectors with higher prior 
achievement (2006 test scores) have higher second-year test scores than students with lower prior achievement. 
Similarly, students who receive exceptional education services also exhibit lower second-year achievement 
compared to non-disabled students. Switching schools, also consistent with prior research (Zimmer et al. 2009; 
Witte and Lavertu 2009) consistently has a negative impact on subsequent student achievement. 
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Table 10. 

Models of Conversion and Non Conversion Independent Charter Impacts 
on Math Achievement, 2006-07 to 2007-08 



Model 


1 


2 


3 




Est. (s.e.) 


Est. (s.e.) 


Est. (s.e.) 


Conversion Charter 2006 


, 170 (. 054 )*** 


, 136 (. 056 )** 


. 114 (. 054 )** 


Non-Conversion Charter 2006 


. 080 (. 061 ) 


, 050 (. 059 ) 


. 034 (. 059 ) 


2006 Score - Math 


, 606 (. 027 )*** 


, 603 (. 024 )**^ 


.601 (. 025 )*** 


2006 Score - Reading 


. 209 (. 021 )*** 


.161 (. 020 )*** 


. 160 (. 020 )*** 


Black 




-. 249 (. 057 )*** 


-. 250 (. 057 )*** 


Female 




. 010 (. 026 ) 


. 007 (. 027 ) 


ExEd 




-. 344 (. 060 )*** 


-. 348 (. 060 )*** 


Free_Lunch 




-. 004 (. 026 ) 


-.001 (. 026 ) 


Switch Sch. 






-. 070 (. 032 )** 


Intercept 


-, 053 (. 022 )** 


. 230 (. 072 )*** 


. 254 (. 072 )*** 




.572 


.587 


.588 


F 


715.27 






N 


3357 


3357 


3357 



***p<0.01, **p<0.05, *p<0.10 

Note: Test scores measured as standardized scores. Models are estimated with robust standard errors. The reference 
category for both conversion charter and non-conversion charter is MPS. For the black indicator variable the 
reference category is non-black which includes whites, Asians, Flispanics and Native Americans. Male is the reference 
category for female. For ExEd, Free_Lunch and Switch School variables the reference categories are not indicating 
this status. Grade level dummies and indicators for missing values of Free_Lunch and ExEd are not included in the 

above estimates. 
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Table 11. 

Models of Conversion and Non Conversion Independent Charter Impacts 
on Reading Achievement, 2006-07 to 2007-08 



Model 


1 


2 


3 




Est. (s.e.) 


Est. (s.e.) 


Est. (s.e.) 


Conversion Charter 2006 


, 124 (. 044 )^** 


. 076 (. 027 )*** 


. 054 (. 028 )* 


Non-Conversion Charter 
2006 


. 003 (. 040 ) 


. 019 (. 038 ) 


. 036 (. 039 ) 


2006 Score -Reading 


.551 (. 023 )*** 


. 499 (. 026 )*** 


. 498 (. 026 )*** 


2006 Score -Math 


. 249 (. 022 )*** 


. 243 (. 020 )*** 


. 240 (. 021 )*** 


Black 




-. 142 (. 053 )*** 


-. 143 (. 054 )*** 


Female 




. 058 (. 025 )** 


. 055 (. 026 )** 


ExEd 




-. 326 (. 058 )*** 


-. 329 (. 058 )*** 


Free_Lunch 




-.111 (. 035 )* 


-. 108 (. 035 )*** 


Switch Sch. 






-. 077 (. 031 )** 


Intercept 


.001 (. 020 )*** 


. 245 (. 059 )*** 


. 272 (. 062 )*** 


R^ 


.576 


.591 


.592 


F 


591.52 






N 


3354 


3354 


3354 



***p<0.01, **p<0.05, *p<0.10 

Note: Test scores measured as standardized scores. Models estimated with robust standard errors. Most of the 
variables included in the models are indicator variables. The reference category for both conversion charter and non- 
conversion charter is MPS. For the black indicator variable the reference category is non-black which includes whites, 
Asians, Flispanics and Native Americans. Male is the reference category for female. For ExEd, Free_Lunch and Switch 
School variables the reference categories are not indicating this status. Grade level dummies and indicators for missing 
values of Free_Lunch and ExEd are not included in the above estimates. 



There are more nuanced results when this main effect is further analyzed hy the type of charter school and 
the distribution of student achievement. These results are depicted in Tables 10 through 13. Some of the 
independent charter schools were conversion schools, being previously private schools in the Milwaukee Parental 
Choice (voucher) Program. Although four out of 10 charter schools were conversion charters, there are many 
fewer students in our sample in these schools. As indicated in Appendix Table A-1, there are only 522 students 
in the conversion charter schools, compared to 1,361 students in the non-conversion charter schools. 

In Table 10 the effect of conversion charter schools on math achievement is positive, compared to our matched 
MPS sample. For example, in model 1 in Table 10 the effect of conversion charter schools on math achievement 
growth, not controlling for student characteristics and switching schools, indicates that students in these 
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schools are performing, on average, .170 standard deviations better than students in MPS schools. Controlling 
for student characteristics and school switching, the effect remains statistically significant and reduces to .114 
standard deviations in model 3 ofTable 10. Students in non-conversion charter schools perform no differently 
in math compared to similar MPS students. 

In reading, students in conversion charter schools appear to be outperforming students in non-conversion 
charter schools and also outperforming students in MPS schools. Specifically, in model 1 ofTable 11 students 
in conversion charters make, on average, . 124 standard deviations more gains than their MPS counterparts 
in reading, without controlling for student characteristics and school switching. When we control for these 
factors the effect reduces to .054 standard deviations but remains statistically significant in model 3 ofTable 11. 
Students in non-conversion charter schools perform the same as their counterparts in MPS. 

When we examine the impact of independent charter schools at different points in the distribution of student 
achievement, we find these charter schools have their strongest positive effects for students starting at baseline 
at the low end of the achievement distribution. This is the result after controlling for student characteristics 
and school switching. Specifically, in math in Table 12, the effects for students at the 25th percentile of the 
achievement distribution are estimated to be .109 greater gains than students in MPS. However, similar 
quantile regressions in reading show charter impacts do not vary for students with different starting levels of 
achievement (Table 13). 



Table 12. 

Results of Quantile Regression for Charter Impacts on Math Achievement 



Math 

(N=3357) 


.10 


.25 


Median 


.75 


.90 


Charter Main Effect 


Charter 


.113(.059)* 


.109(.040)*** 


.046(.033) 


.034(.042) 


.041 (.052) 


Charter Effect Decomposed in Conversion and Non-Conversion Charters 


Conversion-Charter 


.202(.143) 


.151 (.077)* 


.086(.054) 


.040(.071) 


.046(.096) 


Non-Conversion 

Charter 


.090(.085) 


.081 (.055) 


.036(.036) 


.029(.040) 


.037(.060) 



***p<0.01, **p<0.05, *p<0.10 



Note: Test scores measured as standardized scores. Tabled results control for race, free lunch status, exceptional 
education, grade, switching schools and prior achievement. The point estimates for these controls are not included in 
the table but are available upon request. Bootstrapped standard errors are estimated and account for clustering of 

students within schools. 
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Table 13. 

Results of Quantile Regression for Charter Impacts on Reading Achievement 



Reading 

(N= 3,354) 


.10 


.25 


Median 


.75 


.90 


Charter Main Effect 


Charter 


.003 (.065) 


-.013(.050) 


-.023 (.026) 


-.017(.024) 


-.011 (.030) 


Charter Effect Decomposed in Conversion and Non-Conversion Charters 


Conversion-Charter 


.044(.052) 


-.002(.090) 


-.010(.049) 


.017(.035) 


-.009(.046) 


Non-Conversion 

Charter 


-.038(.087) 


-.014(.053) 


-.029(.039) 


-.021 (.042) 


-.021 (.042) 



***p<0.01, **p<0.05, *p<0.10 



Note: Test scores measured as standardized scores. Tabled results control for race, free lunch status, exceptional 
education, grade, switching schools and prior achievement. The point estimates for these controls are not included in the 
table but are available upon request. Bootstrapped standard errors are estimated and account for clustering of students 

within schools. 
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SCHOOL SWITCHING. SECTOR SWITCHING AND STUDY ATTRITION 

School and Sector Switching 

There are a number of students in both MPS and independent charter schools who switched schools from Year 
1 to Year 2. As shown in our models above, switching schools has a negative effect on student achievement, 
thus it is important to take this into account in our estimates and understand the nuances of switching for each 
sample. Students switch schools for different reasons. Some students leave their school for structural reasons, 
meaning they are in a terminal grade and they have no other option but moving to a new school the next year. 
Alternatively, students may leave their school for a non-structural reason like their family is moving or they 
are not satisfied with the school climate and/or quality. A non-structural switch can be modeled by identifying 
students who switch in non-terminal grades. In our analyses, we classify these as two types of within-sector 
switchers. In other words, we identify who switches within sector (students who move from one school to 
another in the same sector) and whether their switch is for a structural or a non-structural reason. In addition, 
we track between-sector switchers, students who began in one sector (charter or MPS) at baseline and switched 
to the other sector in Year 2. Then we identify those students who never made a switch, who are classified as 
non-switchers.^ 



Table 14. 

Switching, by Sector and Type of Switch, 2006-07 to 2007-08 





MPS Matched 

(%) 


Independent 

Charter 

(%) 




1,064 


1,486 


Non Switchers 


(51.9) 


(78.9) 




949*** 


^-|*** 


All Within Sector Switchers: 




(46.3) 


(1.65) 




495*** 


10 


Structural 


(24.1) 


(0.530) 




454 


21 


Non Structural 


(22.1) 


(1.12) 


Between Sector Switchers 


37 


366 


(1.80) 


(19.4) 


Total Non Missing N = 3,933 


2,050 


1,883 



Stars indicate MPS different from Independent Charter statistics at ***p<0.01, **p<0.05, *p<0.10, based on a 
two-taiied t-test. Structurai switchers are students in grade 5 in 2006-07 and grade 6 in 2007-08, as weii as those in 

grade 8 in 2006-07 and grade 9 in 2007-08. 



9 Due to data limitations we cannot explicitly track eighth to ninth grade school switchers in Year 2 so there is an assumption that 
all of these switchers are structural switchers. 
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In Table 14 it is clear that a greater number of students in independent charter schools do not switch compared 
to similar students in MPS. Close to 50 percent of the MPS matched baseline sample switched schools in one 
way or another between November 2006 and November 2007. Part of the reason that there is little within-sector 
switching by charter students is that they are much more likely to switch to MPS schools. While 31 students 
switched charter schools, 366 switched from charter schools to MPS. One of the clear explanations for this 
is that there are fewer charter slots in charter middle and high schools so students have no alternative but to 
switch. In contrast, very few (37 of 2,050) MPS students switched into independent charters during the first 
year. Part of the explanation for this, and the large number of within-sector switching in MPS, is simply that 
there many more school options in MPS than in independent charter schools. 

Study Attrition 

The overall percent of missing students from the study after the first year is 13.6 percent.^ However, as 
indicated in Table 15, there is considerable difference between the two samples, with 9.96 percent missing in 
MPS and 17.3 percent missing from the independent charters. Students in both independent charters and 
MPS may have left for private schools in the MPCP program or may have moved out of the city of Milwaukee. 
We have tracked students between sectors (i.e. “crossovers”) using test score and enrollment data, but we 
undoubtedly missed some students who will be recovered in subsequent years. We do not have the data to track 
students into private schools, which are likely to account for more missing in the charter schools. Given that 
we anticipated sample attrition at 20 percent annually, these results are slightly optimistic. Also as expected, 
missing students are not the same as non-missing students. The relevant data are portrayed in Table 15. Missing 
students are likely to be better students, less likely to be black or on free lunch. Although these differences are 
not large we corrected for these characteristics in our statistical models presented above. 



1 0 Missing from the entire analysis, and not counted in attrition statistics are 36 students from the original 4,590 who had 

nonsensical data for 2007-08. These were students, mostly from two independent charter schools who had implausible grade 
changes from year 1 to 2. Without these students, the proof sample is 4,554. We continue to attempt to correct these data. 
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Table 15. 

Baseline Student Characteristics for Non Missing and Missing Independent Charter and MPS Matched Students 





Non Missing 


Missing 




MPS Matched 


Independent 

Charter 


MPS Matched 


Independent 

Charter 


Female 


1,021 


925 


105 


206 


(%) 


(49.8) 


(49.1) 


(46.2) 


(52.2) 


Black 


2,016*** 


1,773 


220** 


366 


(%) 


(98.3) 


(94.1) 


(96.9) 


(92.8) 


Non Black 


34*** 


110 


y*** 


28 


(%) 


(1.7) 


(5.84) 


(3.1) 


(7.1) 


Free Lunch 


1,518*** 


1,521 


163** 


244 


(%) 


(74.0) 


(80.7) 


(71.8) 


(61.9) 


ExEd 




161 


34*** 


30 


(%) 


(16.2) 


(8.6) 


(14.9) 


(7.6) 


Baseline 


390 


379 


41* 


51 


Grade 3 

(%) 


(19.0) 


(20.1) 


(18.0) 


(12.9) 


Baseline 


388 


366 


32 


51 


Grade 4 

(%) 


(18.9) 


(19.4) 


(14.1) 


(12.9) 


Baseline 


392 


376 


38 


51 


Grade 5 

(%) 


(19.1) 


(19.9) 


(16.7) 


(12.9) 


Baseline 


326 


316 


30 


42 


Grade 6 

(%) 


(15.9) 


(16.7) 


(13.2) 


(10.6) 


Baseline 


305* 


320 


36*** 


26 


Grade 7 

(%) 


(14.8) 


(17.1) 


(15.8) 


(6.6) 


Baseline 


249*** 


126 


50*** 


173 


Grade 8 

(%) 


(12.1) 


(6.7) 


(22.0) 


(43.9) 


TOTAL (N) 


2,050 


1,883 


227 


394 



Stars indicate MPS Matched different from Independent Charter statistics at ***p<0.01, **p<0.05, *p<0.10, 
based on a two-taiied t-test. Percentages are rounded to the tenth decimai point. 
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A more important threat to the inferences permitted hy this study would he if missing students were different 
between MPS and charter students regarding their achievement. As indicated in Table 16, apart from the 
number of missing students (discussed above), there is only one grade on the baseline reading test where missing 
MPS and charter students differ significantly. Thus on the most important control variable, there is essentially 
no difference between the students we were able to track and those we were not. There are more differences on 
student characteristics in the grades in which students are missing (Table 15). However, in these cases, which 
include race, free lunch, and exceptional education, the missing differences mirror the differences in the original 
samples (see Table 4). Thus while attrition did not improve on the initial sample differences, they also did 
not make them worse. As indicated previously, because of the baseline differences in student characteristics, 
we weighted our analysis by the inverse proportions of student characteristics and baseline scores. In the 
multivariate analyses these variables were also used as control variables. 



Table 16. 

Average Baseline Achievement for Non Missing and Missing Independent Charter and MPS Matched Students by Grade 





Non Missing 


Missing 


MPS Matched 


Independent 

Charter 


MPS Matched 


Independent 

Charter 


Grade 3 (N) 


(390) 


(379) 


(41) 


(51) 


Average Math 


384 


387 


371 


376 


Average Reading 


429 


428 


405** 


432 


Grade 4 (N) 


(388) 


(366) 


(32) 


(51) 


Average Math 


416 


422 


404 


420 


Average Reading 


441 


441 


434 


442 


Grade 5 (N) 


(392) 


(376) 


(38) 


(51) 


Average Math 


455 


452 


446 


451 


Average Reading 


451 


448 


451 


441 


Grade 6 (N) 


(326) 


(316) 


(30) 


(42) 


Average Math 


468 


474* 


463 


461 


Average Reading 


465 


460 


453 


453 


Grade 7 (N) 


(305) 


(320) 


(36) 


(26) 


Average Math 


496 


493 


493 


494 


Average Reading 


471 


471 


467 


464 


Grade 8 (N) 


(249) 


(126) 


(50) 


(173) 


Average Math 


505 


503 


514 


511 


Average Reading 


498 


495 


500 


501 


TOTAL (N) 


2,050 


1,883 


227 


394 



Stars indicate MPS Matched different from Independent Charter statistics at ***p<0.01, **p<0.05, *p<0.10, 

based on a two-taiied t-test. 
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Perhaps more important, the differences are driven hy 8th graders who leave the charter schools for high school. 
They account for 43.9 percent of the missing students from the charter sample. These students are undoubtedly 
either going to private high schools or out of district, perhaps using open enrollment. The paucity of high 
school slots in the independent charter schools is driving this untraceahle exodus of sample students. We discuss 
the implications of attrition and possible future analyses in the caveats section below. 



CAVEATS 



There are three issues that affect the explanatory power of these analyses. First, the results in this report are 
based only on the first of four years of estimated achievement gains. Second, as in all studies of urban education 
there are missing data due to sample attrition — 13.6 percent of panelists in both sectors were missing in Year 2. 
Although this number is lower than expected and lower than in a number of other studies, it does raise concerns 
in that both study attrition and between-sample attrition was non-random, most importantly with higher 
achieving baseline students as missing. This could affect accurate overall population estimates of gains, but 
because there were almost no baseline test differences between missing students from either sample, we believe 
our sample comparisons are accurate and should remain so. We nevertheless weighted for these differences 
accordingly, and in future analyses we could perform a number of analyses in addition to the full sample analyses 
done here. For example, we could eliminate from the study the paired students of missing students. Another 
way to control for attrition as well as crossovers would be to limit the samples to only students who are not 
missing and in the same sectors (i.e. non-crossovers) in all years. The latter approach was recently employed 
in the longitudinal study of the Milwaukee voucher program (Witte et al. 2010). Thus we feel confident that 
attrition can be handled well in future years even if the non-random nature of that attrition becomes worse. 
Finally, the findings about the performance of students who are non-black are limited because of the very small 
numbers of non-blacks in the sample. These racial groups that are enrolled in independent Milwaukee charter 
schools in very small numbers include whites, Hispanics, Asians, and Native Americans. 
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SUMMARY AND CONCLUSIONS 



This report is the first of three reports about the performance of independent charter students in Milwaukee 
compared to a matched sample of students in the Milwaukee Public Schools. The results of one year of 
achievement growth are mixed. Using regression models that produce the most precise estimates of 2007 
achievement, when we compare students in all independent Milwaukee charters to matched MPS students 
there are few significant effects of attending a charter school on achievement gains in either math or reading. 

The exception is in one of our three models for mathematics gains. When we control for prior achievement, 
and not for student characteristics or switching schools, students in charter schools gain approximately .105 
standard deviations more than students in MPS. There are no differences in any models in reading. An analysis 
of differences within charter schools provides a more nuanced picture. Students in conversion (from private) 
independent charter schools perform better than their MPS counterparts in both math and reading after 
controlling for student characteristics and school switching. Further for mathematics, but not reading, the gains 
were for students with baseline scores near the bottom of the initial achievement distribution. Because students 
in conversion charters are only approximately 13.3 percent of all charter students in our analytic sample, these 
results were not enough to make the general result statistically significant. 

We also analyzed school and sector switching and attrition from our study. Although considerable school 
switching occurred between 2006 and 2007, there was more overall switching among MPS students. That is 
one reason we control for that switching in estimating our models. The switching in the two samples was very 
different, however. In MPS almost all the switching was between MPS schools. For charter students almost 
all the switching was sector switching to MPS schools. We suspect this is because charter students going into 
middle and high schools had many fewer options in charter schools than in MPS schools. 

Attrition was relatively modest at approximately 13.6 percent of the Year 1 sample, with higher numbers of 
missing students in charter schools. Overall attrition was not random, with better students from higher socio- 
economic status families more likely to be missing. There was also some non-random attrition between sectors, 
but not in baseline test scores which was our primary concern. The differences in student characteristics between 
charter and MPS matched students resulted from differences in the original 2006 samples. We adjusted for 
these differences with weights and control variables in our multivariate analysis. 

As indicated in the Executive Summary and throughout the analysis, we caution that the results in this report 
are based only on the first of four years of estimated achievement gains. Subsequent reports may well alter the 
general findings and conclusions. 
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APPENDIX 



Table A-1: 

Descriptive Statistics for Variables in the Achievement Model 





Conversion 

Charter 


(%) 


Non- 

Conversion 

Charter 


(%) 


MPS 

Matched 


(%) 


Black 


434 


(83.1)*** 


1,339 


(98.4) 


2,016 


(98.3) 


Female 


266 


(51.0) 


659 


(48.4) 


1,021 


(49.8) 


ExEd 


58 


(11.1)*** 


103 


(7.57)*** 


332 


(16.2) 


Eree_Lunch 


307 


(58.8)*** 


1,214 


(89.2)*** 


1,518 


(74) 


Switch School 


58 


(11.1)*** 


339 


(24.9)*** 


986 


(48.1) 


School Type N 


522 


- 


1,361 


- 


2,050 


3,933 TOTAL 



Note: The columns Conversion Charter, Non-Conversion Charter and MPS Matched contain the N for each variable in the 
achievement model. School Type N is not a total of these columns; rather it is the total N for each school type. 



***Different from MPS Matched at p<0.01, **Different from MPS Matched at p<0.05, 
*Different from MPS Matched at p<0.1 0, two-tailed test. 
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